The Utility of Randomness in Decision Tree Ensembles

The use of randomness in constructing decision tree ensembles has drawn much attention in the machine learning community. In general, ensembles introduce randomness to generate diverse trees and in turn they enhance ensembles’ predictive accuracy. Examples of such ensembles are Bagging, Random Forests and Random Decision Tree. In the past, most of the random tree ensembles inject various kinds of randomness into deterministic models. Very few of these ensembles considered variable randomness or found it insensitive to performance. In contrast, this thesis uses complete-random tree ensembles as a starting point to investigate the utility of randomness and finishes with a variable random model, capable of finding the appropriate settings of randomness for individual data sets in order to improve predictive accuracy. Firstly, we construct a taxonomy of tree randomisations to categorise existing randomisation techniques. Then, we analyse the benefits and problems of different randomisation techniques to gain a better understanding of their effects. Secondly, we find that the key component of random tree ensembles is simply the probability averaging ensemble method. Based on the results of vigorous experimentations, probability averaging brings out the best of complete-randomness in decision tree ensembles. Using this key component alone permits the highest degree of diversity. We name this complete-random tree algorithm Max-diverse Ensemble as it achieves exceptional accuracy by maximising diversity. Interestingly, without the presence of any feature selection criterion, Max-diverse Ensemble’s accuracy is comparable to Random Forests, a popular implementation of random tree ensemble. Furthermore, visual evidence shows that complete-randomness provides a distinctive representational power to model target concepts. Taking the advantages of this representational power, we propose a decision tree algorithm with variable randomness. It is called Max-diverse.α. Max-diverse.α forms a smooth convex error-rate contour using different degrees of randomness. In many cases, individual data sets with appropriate settings of randomness achieve better accuracy than the complete-random settings. Finally, we propose a simple estimation technique for estimating an effective settings of randomness generated entirely from the progressive training errors. Applying the estimation technique, Max-diverse.α improves significantly from Max-diverse Ensemble. The experimental results show that Max-diverse.α performs significantly better than Random Forests and comparably with the state-of-the-art C5 Boosting.